Dynamic Loading IB verbs by AtlantaPepsi · Pull Request #253 · ROCm/TransferBench

AtlantaPepsi · 2026-04-08T06:06:14Z

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

* Adding support for GFX/DMA executors accessing remote memory via UALoE

* CUDA Driver API addition * NVML initiation * MNNVL support * pod presets

…eck issue for VMs

merge develop changes

* fix qpCount storage limit to allow 256+ (ROCm#237) * fix function header GetClosestGpusToNic (ROCm#238) fix the function header GetClosestGpusToNic to match the function definition and function calls * Fixed CQ size for high QPs cases and poll CQ in batch CQ Size: max(100, qpCount) - dynamically sized This avoid hangs at large QPs size, notably experienced with small message size (ex: 256 QPs, 8M message size) Polling: Up to 32 completions per poll call to reduce poll calls * improve DMABUF zcat check improve DMABUF zcat check, similar to ROCM-2855 * add NIC_CQ_POLL_BATCH option as CQ poll batch size Add NIC_CQ_POLL_BATCH as an option to ibv_poll_cq for CQ poll batch size set a default value to `4` which appears to be current RCCL default replace fixed wc_array with vector wc.data Files changed: - `src/header/TransferBench.hpp` - `src/client/EnvVars.hpp` * align with develop * wc_array move out of the while loop from PR review * Update CHANGELOG.md * Revert "fix function header GetClosestGpusToNic (ROCm#238)" This reverts commit a8cf384. * Revert "improve DMABUF zcat check" This reverts commit 6d88473. --------- Co-authored-by: Pak Nin Lui <pak.lui@amd.com>

Copilot

Pull request overview

This PR introduces an IBV_DIRECT build/runtime switch intended to support resolving libibverbs symbols either via direct linkage or via dlopen/dlsym, and adds a runtime check to block NIC executor usage when verbs aren’t available.

Changes:

Added IBV_DIRECT mode plumbing (Makefile + CMake option) to toggle direct symbol binding vs dlsym resolution.
Introduced pfn_* function pointers and updated some verbs call sites/macros to call through them.
Added System::IbvLoaded() and a guard to error out when NIC executor is requested but verbs aren’t loaded.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 9 comments.

File	Description
src/header/TransferBench.hpp	Adds libibverbs dynamic-loading infrastructure (`pfn_`, `Ibvdl()`), runtime loaded-state tracking, and uses `pfn_` in some verbs calls.
Makefile	Adds `DISABLE_IBV_DIRECT` flag and sets `-DIBV_DIRECT=1` by default when NIC exec is enabled.
CMakeLists.txt	Adds `ENABLE_IBV_DIRECT` option and defines `IBV_DIRECT=1` when enabled.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-08T06:19:56Z

+namespace {
+  IBV_FN(ibv_alloc_pd, ibv_pd*, (ibv_context*));
+  IBV_FN(ibv_close_device, int, (ibv_context*));
+  IBV_FN(ibv_create_cq, ibv_cq*, (ibv_context*, int, void*, ibv_comp_channel*, int));
+  IBV_FN(ibv_create_qp, ibv_qp*, (ibv_pd*, ibv_qp_init_attr*));
+  IBV_FN(ibv_dealloc_pd, int, (ibv_pd*));
+  IBV_FN(ibv_dereg_mr, int, (ibv_mr*));
+  IBV_FN(ibv_destroy_cq, int, (ibv_cq*));
+  IBV_FN(ibv_destroy_qp, int, (ibv_qp*));
+  IBV_FN(ibv_free_device_list, void, (ibv_device**));
+  IBV_FN(ibv_get_device_list, ibv_device**, (int*));
+  IBV_FN(ibv_get_device_name, const char*, (ibv_device*));
+  IBV_FN(ibv_modify_qp, int, (ibv_qp*, ibv_qp_attr*, int));
+  IBV_FN(ibv_open_device, ibv_context*, (ibv_device*));
+  IBV_FN(ibv_poll_cq, int, (ibv_cq*, int, ibv_wc*));
+  IBV_FN(ibv_post_send, int, (ibv_qp*, ibv_send_wr*, ibv_send_wr**));
+  IBV_FN(ibv_query_device, int, (ibv_context*, ibv_device_attr*));
+  IBV_FN(ibv_query_gid, int, (ibv_context*, uint8_t, int, ibv_gid*));
+  IBV_FN(ibv_query_port, int, (ibv_context*, uint8_t, ibv_port_attr*));
+#ifdef HAVE_DMABUF_SUPPORT
+  IBV_FN(ibv_reg_dmabuf_mr, ibv_mr*, (ibv_pd*, uint64_t, size_t, uint64_t, int, int));
+#endif
+  IBV_FN(ibv_reg_mr, ibv_mr*, (ibv_pd*, void*, size_t, int));
+}


The new IBV_FN() function-pointer declarations are not guarded by NIC_EXEC_ENABLED, but IBV_FN (and ibv_* types) only exist when NIC_EXEC_ENABLED includes <infiniband/verbs.h>. As-is, builds with NIC executor disabled will fail to compile at this block. Wrap the IBV_FN declarations (and any dependent IBV_* call macros) in #ifdef NIC_EXEC_ENABLED, or provide a no-op/alternative definition when NIC support is off.

Copilot · 2026-04-08T06:19:56Z

+  // Should only be called with IBV_DIRECT guard
+  static void* Ibvdl() {
+    static void* ibvLibHandle = nullptr;
+    if (ibvLibHandle) return ibvLibHandle;
+
+    void *handle = dlopen("libibverbs.so.1", RTLD_NOW);
+    if (handle != nullptr) {


Ibvdl() uses dlopen/dlsym/dlclose but is compiled unconditionally under NIC_EXEC_ENABLED. When IBV_DIRECT=1, <dlfcn.h> is not included, so these calls will be undeclared and compilation will fail. Guard Ibvdl() itself (and any dl* usage) with #if !IBV_DIRECT (or include <dlfcn.h> regardless of IBV_DIRECT).

Copilot · 2026-04-08T06:19:56Z

+      for (size_t i = 0; i < sizeof(symbols) / sizeof(symbols[0]); i++) {
+        *symbols[i].ppfn = dlsym(handle, symbols[i].name);
+        if (*symbols[i].ppfn == nullptr) {
+          // Log("[WARN] Failed to load symbol %s", symbols[i].name);
+          dlclose(handle);
+          break;
+        }
+      }


Ibvdl() currently treats partial symbol resolution as success: on a dlsym failure it dlclose()s the handle but still leaves handle non-null and later assigns it to the cached ibvLibHandle. This can return a closed handle and leave some pfn_* pointers non-null/invalid. Only mark the library loaded if all symbols are resolved; otherwise set handle to nullptr and reset any already-assigned function pointers.

Suggested change

for (size_t i = 0; i < sizeof(symbols) / sizeof(symbols[0]); i++) {

*symbols[i].ppfn = dlsym(handle, symbols[i].name);

if (*symbols[i].ppfn == nullptr) {

// Log("[WARN] Failed to load symbol %s", symbols[i].name);

dlclose(handle);

break;

}

}

auto resetSymbols = [&symbols]() {

for (size_t i = 0; i < sizeof(symbols) / sizeof(symbols[0]); i++) {

*symbols[i].ppfn = nullptr;

}

};

bool loadedAllSymbols = true;

for (size_t i = 0; i < sizeof(symbols) / sizeof(symbols[0]); i++) {

*symbols[i].ppfn = dlsym(handle, symbols[i].name);

if (*symbols[i].ppfn == nullptr) {

// Log("[WARN] Failed to load symbol %s", symbols[i].name);

loadedAllSymbols = false;

break;

}

}

if (!loadedAllSymbols) {

resetSymbols();

dlclose(handle);

handle = nullptr;

}

Copilot · 2026-04-08T06:19:57Z

    static vector<IbvDevice> ibvDeviceList = {};

+#if !defined(IBV_DIRECT)
+    if (ibvLibHandle == nullptr) {


GetIbvDeviceList() checks ibvLibHandle == nullptr, but ibvLibHandle here refers to no visible variable (the System member isn’t accessible from this free function, and the Ibvdl() local static isn’t in scope). This is a compile error. Use a reachable source of truth (e.g., System::Get().IbvLoaded(), store the handle in a shared internal static with external linkage in this anonymous namespace, or call Ibvdl() and cache its result) instead of referencing an out-of-scope identifier.

Suggested change

if (ibvLibHandle == nullptr) {

if (Ibvdl() == nullptr) {

Copilot · 2026-04-08T06:19:57Z

+    if (ibvLibHandle) {
+      dlclose(ibvLibHandle);
+      ibvLibHandle = nullptr;
+    }


System::~System() calls dlclose(ibvLibHandle) unconditionally. When IBV_DIRECT=1, <dlfcn.h> is not included, so dlclose will be undeclared and compilation can fail. Also, Ibvdl() keeps its own internal static handle, so closing only the System member can leave Ibvdl() returning a stale (already-closed) handle on future calls. Guard dlclose with #if !IBV_DIRECT and keep a single owner for the handle (either System or Ibvdl, but not both).

Suggested change

if (ibvLibHandle) {

dlclose(ibvLibHandle);

ibvLibHandle = nullptr;

}

#ifdef NIC_EXEC_ENABLED

#if !IBV_DIRECT

// Ibvdl() owns the dynamically loaded libibverbs handle. Avoid closing a

// cached copy here so future Ibvdl() calls cannot observe a stale handle.

ibvLibHandle = nullptr;

#endif

#endif

Copilot · 2026-04-08T06:19:57Z

      LDFLAGS += -libverbs
      NIC_ENABLED = 1
+      ifneq ($(DISABLE_IBV_DIRECT),1)
+        COMMON_FLAGS += -DIBV_DIRECT=1


When DISABLE_IBV_DIRECT=1 (dlsym path), the build still unconditionally links -libverbs. Any remaining direct ibv_* calls (and even an unused -libverbs without --as-needed) can keep libibverbs as a hard runtime dependency, undermining the goal of optional runtime loading. Consider omitting -libverbs when using the dlsym path (and ensure all ibv_* usage goes through pfn_*), and add -ldl for toolchains/glibc versions where dlopen is not in libc.

Suggested change

LDFLAGS += -libverbs

NIC_ENABLED = 1

ifneq ($(DISABLE_IBV_DIRECT),1)

COMMON_FLAGS += -DIBV_DIRECT=1

NIC_ENABLED = 1

ifneq ($(DISABLE_IBV_DIRECT),1)

COMMON_FLAGS += -DIBV_DIRECT=1

LDFLAGS += -libverbs

else

LDFLAGS += -ldl

Copilot · 2026-04-08T06:19:58Z

  target_link_libraries(TransferBench PRIVATE ${IBVERBS_LIBRARY})
  target_compile_definitions(TransferBench PRIVATE NIC_EXEC_ENABLED)
+  if(ENABLE_IBV_DIRECT)


ENABLE_IBV_DIRECT=OFF claims to resolve symbols via dlsym at runtime, but the target still links against ${IBVERBS_LIBRARY} unconditionally when IBVERBS_FOUND. This keeps libibverbs as a hard dependency and defeats optional loading. If runtime optionality is intended, avoid linking ${IBVERBS_LIBRARY} when ENABLE_IBV_DIRECT is OFF and ensure all ibv_* usage is through the loaded function pointers.

Suggested change

target_link_libraries(TransferBench PRIVATE ${IBVERBS_LIBRARY})

target_compile_definitions(TransferBench PRIVATE NIC_EXEC_ENABLED)

if(ENABLE_IBV_DIRECT)

target_compile_definitions(TransferBench PRIVATE NIC_EXEC_ENABLED)

if(ENABLE_IBV_DIRECT)

target_link_libraries(TransferBench PRIVATE ${IBVERBS_LIBRARY})

Copilot · 2026-04-08T06:19:58Z

      // Query the number of IBV devices
      int numIbvDevices = 0;
-      ibv_device** deviceList = ibv_get_device_list(&numIbvDevices);
+      ibv_device** deviceList = pfn_ibv_get_device_list(&numIbvDevices);



Within GetIbvDeviceList(), most verbs calls were converted to pfn_* (e.g., pfn_ibv_get_device_list/pfn_ibv_open_device/etc.), but the function still calls ibv_free_device_list(deviceList) directly later in the same function. In the dlsym path this reintroduces a direct libibverbs symbol reference and breaks the “all calls through pfn_*” model. Switch that cleanup call to pfn_ibv_free_device_list as well.

Copilot · 2026-04-08T06:19:58Z

+  // Should only be called with IBV_DIRECT guard
+  static void* Ibvdl() {


The comment above Ibvdl() says it “should only be called with IBV_DIRECT guard”, but Ibvdl() is the dlsym/dlopen path and should only be used when IBV_DIRECT is disabled. Update the comment (and ideally add a compile-time guard) to avoid confusion about which mode this helper supports.

gilbertlee-amd and others added 20 commits February 19, 2026 17:32

Initial pod communication support (ROCm#235)

0252788

* Adding support for GFX/DMA executors accessing remote memory via UALoE

Adjust min HIP version in Makefile for pod support

9edaae8

Adding TB_DUMP_CFG_FILE and fixing a deallocation bug

2b17e62

Adding gfx1250 to CMakeFiles

2b707b7

Adjusting how HIP headers are included

2bb8302

Updating a2asweep and scaling presets

060abc2

Fixing logging to prevent recursive error

6c2ecf7

Fixing fabric handle bug

794bcf7

Changing table formatting to make it easier to paste

8de0154

Showing num iterations when running in timed mode

4a0f390

cuda + MNNVL update & pod presets (ROCm#241)

bf49ba4

* CUDA Driver API addition * NVML initiation * MNNVL support * pod presets

Changing NIC_FILTER to TB_NIC_FILTER

5e61666

prefixing remaining env vars with TB_, fixing potential filesystem ch…

bec2c5e

…eck issue for VMs

Fixing TB_PAUSE issue

561e2f7

Merge pull request ROCm#245 from ROCm/develop

94cf3c9

merge develop changes

fix hang when NVML is present but fabricmanager isnt (ROCm#246)

275998b

Adding HBM read bandwidth preset (ROCm#250)

168cdc1

Adding TB_WALLCLOCK_RATE in case wallclock rate is reported as 0

fdec7d5

ibv dynamic loading

94e581d

nileshnegi requested a review from Copilot April 8, 2026 06:10

Copilot started reviewing on behalf of nileshnegi April 8, 2026 06:12 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

ibv dynamicloading; addition of CUMEM flag and compilation fixes

ebde17f

AtlantaPepsi changed the base branch from candidate to merge/TransferBench-v1.67.0 April 27, 2026 06:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamic Loading IB verbs#253

Dynamic Loading IB verbs#253
AtlantaPepsi wants to merge 21 commits intoROCm:merge/TransferBench-v1.67.0from
AtlantaPepsi:ibdl

AtlantaPepsi commented Apr 8, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		// Should only be called with IBV_DIRECT guard
		static void* Ibvdl() {

Conversation

AtlantaPepsi commented Apr 8, 2026

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants